System for highlighting targets on head up displays with near focus plane

ABSTRACT

Methods and apparatus for displaying a near-focus one-dimensional marker on a head up display (HUD) such that the parallax disparity between the marker and a distant far-focus visual target is not apparent to a viewer looking at the visual target through the HUD. The one-dimensional marker is oriented at the same angle as the interocular line of the viewer&#39;s eyes, as captured by a digital camera facing the viewer. Additionally, a dashed one-dimensional marker is disclosed to reduce the visual noticeability of small inaccuracies in orientation.

BACKGROUND

A head up display (HUD) for a viewer in a vehicle or comparableenvironment provides a transparent display through which the viewer seesvisual targets in the environment. The HUD presents visual informationand/or identifying markers which the viewer can see without having todivert the eyes from the visual targets seen through the HUD. This isparticularly valuable when the viewer is a driver, pilot, co-pilot,navigator, crane operator, or other person with controlresponsibilities.

In some HUDs the visual information and/or identifying markers arevirtual images, such as those reflected off the rear surface of atransparent pane (e.g., a vehicle windshield). In other HUDs the visualinformation and/or identifying markers are not images but rather arereal object displays, such as those produced by a thin electronicdisplay (e.g., a transparent liquid-crystal display).

For HUDs that produce real object displays, the objects are typically inthe near-focus plane, in contrast to the visual targets seen through theHUD, which are typically in the far-focus plane. This is also the casefor a subset of virtual image HUDs, where the reflected image is in thenear focus plane. Near-focus HUDs thus introduce a parallax disparitywith respect to the visual targets, as described below.

For an observed far-focus visual target, human binocular vision divergesthe eyes according to the distance of the visual target, in accordancewith the parallax of the visual target. A near-focus HUD highlightingmarker, however, has a substantially different parallax from that of thefar-focus visual target, resulting in a double image for the marker inthe perceived unified binocular image, to which the viewer may notreadily adapt, thereby reducing the visual effectiveness of thenear-focus HUD. Currently, the only way to avoid this problem is toutilize HUDs with a reflected image in the far-focus plane, but this mayrestrict the HUD to span only a relatively small portion of the visualfield, and may not be practical to use in certain situations. Inaddition, this may rule out the use of HUDs that utilize real objectdisplays, which are typically in the near-focus plane.

SUMMARY

Embodiments of the present invention provide a one-dimensionalnear-focus HUD highlighting marker which extends parallel to theinterocular line of the viewer's eyes, and thereby avoids the doubleimage resulting from the parallax disparity.

In embodiments of the invention, one-dimensional markers extend linearlyin the two-dimensional space of a near-focus HUD, with only enoughthickness to be readily visible.

In an embodiment of the invention, solid straight lines are used as theone-dimensional highlighting markers to circumvent the double imageproblem.

In another embodiment of the invention, dashed straight lines are usedas the one-dimensional highlighting markers, to further reduce thevisual effect of residual double images caused by minor misalignment ofthe orientation of the highlighting markers with the interocular line ofthe viewer's eyes.

Therefore, according to an embodiment of the invention, there isprovided a method for displaying, to a viewer, a marker on a head updisplay (HUD), the method including: measuring a viewer interocular lineorientation relative to a predetermined reference line of the HUD; anddisplaying a one-dimensional marker having a marker orientation anglemeasured relative to the predetermined reference line of the HUD,wherein the marker orientation angle is substantially equal to theviewer interocular orientation angle.

In addition, according to another embodiment of the invention, there isprovided an apparatus for controlling a head up display system, fordisplaying, to a viewer, a marker on a head up display (HUD), theapparatus including: a visual target identifier, for identifying aposition of a visual target which the viewer is viewing through the HUD;a digital camera, for imaging the viewer's face; an eye location unit,for locating respective positions of the viewer's eyes; an interocularline orientation angle unit, for measuring the orientation angle of theviewer's eyes interocular line relative to a reference line of the HUD;and a marker display generator for controlling the HUD system to displaya one-dimensional marker according to the position of the visual target,wherein an orientation angle of the one-dimensional marker relative tothe reference line of the HUD is substantially equal to the orientationangle of the internocular line of the viewer's eyes relative to thereference line.

BRIEF DESCRIPTION OF THE DRAWINGS

The subject matter disclosed may best be understood by reference to thefollowing detailed description when read with the accompanying drawingsin which:

FIG. 1 illustrates the double image effect for a typical prior artnear-focus HUD highlighting marker.

FIG. 2A and FIG. 2B illustrate aligning a one-dimensional HUDhighlighting marker in parallel with the interocular line of theviewer's eyes, according to embodiments of the invention.

FIG. 3A illustrates a residual misalignment of one-dimensional line HUDmarkers.

FIG. 3B illustrates compensating for residual alignment by using dashedone-dimensional line HUD markers according to an embodiment of theinvention.

FIG. 3C illustrates geometric parameters of the dashed one-dimensionalline HUD markers of FIG. 3B.

FIG. 4 illustrates a method for displaying a head up display markeraccording to embodiments of the invention.

FIG. 5 illustrates a conceptual block diagram of apparatus fordisplaying a head up display marker according to embodiments of theinvention.

For simplicity and clarity of illustration, elements shown in thefigures are not necessarily drawn to scale, and the dimensions of someelements may be exaggerated relative to other elements. In addition,reference numerals may be repeated among the figures to indicatecorresponding or analogous elements.

DETAILED DESCRIPTION

FIG. 1 illustrates the double-imaging problem with a prior-artnear-focus HUD marker. A viewer's right eye 107 and left eye 109 havingan interocular distance 108 are focused on a visual target 101 through awindshield 103, on whose rear surface a reflected near-focus HUD marker105 is positioned to indicate visual target 101. It is noted that inthis non-limiting example HUD marker 105 is optically a virtual image,and because it is reflected off windshield 103, it will appear to bevisually farther away than windshield 103.

Visual target 101 is far-focus, but HUD marker is near-focus. Because ofa disparity between the parallax of visual target 101 and HUD marker 105with respect to interocular distance 108, right eye 107 sees HUD marker105 to the left of visual target 101 in a retinal image 111, whereasleft eye 109 sees HUD marker 105 to the right of visual target 101 in aretinal image 113. Thus, when retinal images 111 and 113 are visuallyintegrated, a combined perceived image 115 shows a single image 117 ofvisual target 101, but double images 119 and 121 of HUD marker 105.

FIG. 2A illustrates the effects of an orientation of a viewer's head 201resulting in an interocular line 203 of the viewer's eyes, which isoriented relative to a predetermined reference line associated with theHUD. In an embodiment of the invention, the HUD has a predefinedreference line 209, and a secondary reference line 207 is parallel toreference line 209 of the HUD. In certain embodiments of the invention,predefined reference line 209 is horizontal relative to the vehicle. Inanother embodiment, secondary reference line 207 is parallel to an axisof a predefined coordinate system 225 associated with the HUD. In afurther embodiment, secondary reference line 207 is adjusted to passthrough the viewer's line-of-sight, and may be used as a reference lineto measure the angle of the interocular line 203. According toembodiments of the invention, a one-dimensional reflected HUD marker 211is substantially parallel to interocular line 203, which is notnecessarily parallel to reference line 207.

FIG. 2B illustrates the above relationships from the viewer's viewpoint,and also shows visual target 101 through windshield 205. HUD marker 211is parallel to the interocular line 203, which is not necessarilyparallel to reference line 207. As noted above, predeterminedtwo-dimensional coordinate system 225 can equivalently be defined forthe HUD, and orientation of interocular line 203 may be measuredaccording to an angle relative to coordinate system 225. Even if notexplicitly defined, a predetermined two-dimensional coordinate system isimplicitly defined by a line reference, such as predefined referenceline 209. The reference axes and reference lines shown in FIG. 2A andFIG. 2B need not be displayed in the HUD, however, and in certainembodiments of the invention, the reference axes and reference lines arenot seen by the viewer.

HUD marker 211 is in the near-focus plane, whereas visual target 101 isin the far-focus plane, and therefore when the viewer's eyes are fixedon visual target 101, there necessarily must be a binocular visionparallax disparity that causes a double image of the HUD marker 211.However, because HUD marker 211 is one-dimensional and oriented parallelto interocular line 203, HUD marker 211 has translational symmetry sothat the double images thereof are visually indistinguishable and appearto the viewer as a single image rather than as a double image. A segment213 of HUD marker 211 is visualized by the viewer's left eye, and asegment 215 of HUD marker 211 is visualized by the viewer's right eye. Asegment 217 of HUD marker 211 is visualized by both eyes.

In the example of the invention illustrated in FIG. 2B, HUD marker 211is superimposed directly over the visual center of visual target 101 asshown.

Measuring orientation of the interocular line 203 involves onlymeasuring the angle of the straight line connecting the viewer's eyesrelative to a reference coordinate of the HUD projection apparatus. Inexamples of the invention, the HUD projection apparatus has a fixedpredetermined orientation relative to the windshield, which in turnmeans that measuring orientation of the interocular line 203 involvesonly measuring the angle of the straight line connecting the viewer'seyes relative to windshield horizontal reference 207. This may beaccomplished by known methods and apparatus which tracks the positionsof the viewer's eyes (as discussed below). Although the eye positionsmust be tracked, this does not require tracking the gaze of the viewer(i.e., identifying what the viewer is looking at).

Avoidance of Non-Horizontal Marker Components

According to embodiments of the invention, it is preferable to avoidusing markers that have an observable geometric component perpendicularto the viewer's interocular line. In other words, if the viewer'sinterocular line is considered “horizontal”, then markers having a“vertical” component should be avoided. According to embodiments of theinvention as detailed below, small deviations from the horizontal can betolerated, but markers that are noticeably off-horizontal (e.g.,diagonal, vertical) should be avoided.

In a similar manner, markers that do not have a sufficient horizontalextent, as detailed below, (e.g., a dot) should also be avoided.

Design of Markers to Account for Small Misalignments

Even though methods and apparatus for tracking the viewer's eyepositions are well-established, it is possible that small inaccuraciesin measuring the viewer's interocular line 203 may occur, and may thusresult in small misalignments of HUD marker 211. A small misalignment inturn may result in a small vertical separation of the left- andright-eye visualizations of HUD marker 211, which may be perceived bythe viewer as a double-vision bluffing of HUD marker 211.

According to an example of the invention, the HUD marker can be modifiedto account for a small misalignment, so that the misalignment is notvisually apparent to the viewer.

FIG. 3A illustrates the effect of a small orientation misalignment ofHUD markers 305 and 307 as reflected off a windshield 303. In thisnon-limiting example, the viewer's head (not shown) is tilted so thatthe interocular line of the viewer's eyes is parallel to the lines 304and 306 on windshield 303. However, HUD markers 305 and 307 respectivelyare slightly misaligned in orientation, as shown in FIG. 3A.

As before, viewer's eyes 309 and 311 are on visual target 101, so thatin a combined binocular image 313, image 315 appears as a single imageof visual target 101. Because of the slight misalignment in orientationof HUD markers 305 and 307 relative to interocular line parallels 304and 306, respectively, HUD marker 305 is imaged as a double image withan image 305L from left eye 311 and with an image 305R from right eye309. Likewise, HUD marker 307 is imaged as a double image with an image307L from left eye 311 and an image 307R from right eye 309. Whenvisually processed by a low-level perceptual process 317, a perceivedimage 319 has the same double images of HUD markers 305 and 307, i.e.,images 305L and 305R for HUD marker 305; and images 307L and 307R forHUD marker 307.

FIG. 3B illustrates the use of HUD markers 325 and 327, which have thesame degree of slight misalignment as HUD markers 305 and 307 (FIG. 3A),but which are perceived differently, without the double image. Thedifference in perception is on account of the design of HUD markers 325and 327, which comprise broken segments rather than a continuous whole.As in FIG. 3A, because of the slight misalignment in orientation of HUDmarkers 325 and 327 relative to interocular line parallels 304 and 306,respectively, HUD marker 325 is imaged as a double image with an image325L from left eye 311 and with an image 325R from right eye 309.Likewise, HUD marker 327 is imaged as a double image with an image 327Lfrom left eye 311 and an image 327R from right eye 309. However,provided that the misalignment in orientation is small, in a perceptualprocess 329 the broken segments of images 325R and 325L, and of images327R and 327L may be subject to the gestalt effect, whereby the braininterprets images 325R and 325L as a whole rather than as disconnectedparts, and images 327R and 327L likewise as a whole. Thus, a perceivedimage 339 has single images of HUD markers 335 and 337, without thedouble image effect.

FIG. 3C illustrates the geometric parameters of segmented HUD marker 325which must be properly adjusted in order for image 325R and 325L to beperceived as a single image in spite of the small misalignment.

With reference to FIG. 1, it is seen that parallax disparity of visualtarget 101 with respect to HUD marker 105 results from the differentparallax of near-focus placement of HUD marker 105 from that offar-focus visual target 101, this disparity equals parallax disparitydistance 155, as determined by triangles 151 and 153, which share acommon base of interocular distance 108. Triangle 151 has sides pointingto visual target 101, and triangle 153 has sides pointing to HUD marker325. Parallax disparity distance 155 (in FIG. 1) represents the width oftriangle 151 at the visual distance of HUD marker 105.

A special case of interest occurs when visual target 101 (FIG. 1) is atan optical “infinity” point—where visual target 101 is sufficiently faraway that the distance to visual target 101 is so much greater thaninterocular distance 108 that the sides of triangle 151 can beapproximated as being substantially parallel. In the limiting case asthe distance to visual target 101 goes to infinity, parallax disparitydistance 155 approaches interocular distance 108. In such cases,parallax disparity distance 155 can be reasonably approximated as equalto intraocular distance 108 when the viewer is looking straight ahead.If the viewer's head is turned while the viewer continues to lookstraight ahead, intraocular distance 108 is appropriately foreshortened.In most cases of interest, however, intraocular distance 108 can beapproximated as being constant.

According to an embodiment of the invention, a marker dash-to-dashdistance 351 (in FIG. 3C) equals the distance from the start of a dash357 to the start of an adjacent dash 359, and marker dash-to-dashdistance 351 is set equal to parallax disparity distance 155 (in FIG. 1)divided by an arbitrary integer n, i.e., marker dash-to-dash distance351 is a unit fraction (1/n) of parallax disparity distance 155.Arbitrary integer n may be chosen to optimize the visual appearance ofdashed lines 335 and 337. In a non-limiting example shown in FIG. 3C,n=2.

A dash segment length 353 and a dash gap length 355 may be setarbitrarily provided that their sum equals marker dash-to-dash distance351. In a non-limiting example of the invention, dash segment length 353and dash gap length 355 are equal. In alternative representations, dashsegment length 353 and dash gap length 355 may be represented as a ratioof one to the other or individually as percentages of markerdash-to-dash distance 351.

The setting of the above parameters is similar for segmented HUD marker327.

In the example of the invention illustrated in FIG. 3B, HUD markers 325and 327 are superimposed above and below, respectively, the top andbottom boundaries of visual target 101 as shown. In an embodiment of theinvention, the top and bottom boundaries of visual target 101 are thetop and bottom boundaries, respectively, of a bounding box 341 forvisual target 101.

Method

FIG. 4 illustrates a method for displaying a head up display (HUD)marker according to an embodiment of the invention. In a step 451 avisual target is detected, and the three-dimensional location of thevisual target is stored as a target 3D location 453. Next, in atarget-to-HUD projection step 455, a target 2D location 457 is stored.When it is desired to highlight the target with a marker according toembodiments of the invention, a step 405 measures a “horizontal”orientation relative to a reference line of the HUD and stores themeasured value as a horizontal orientation angle 403. In subsequentsteps a one-dimensional HUD marker is displayed with the same (orsubstantially the same) orientation angle as horizontal orientationangle 403. In a related embodiment the “horizontal” orientation isdetermined by measuring an interocular line 203 (FIG. 2A) relative to apredetermined reference line (such as reference line 207 in FIG. 2A);and in this embodiment horizontal orientation angle 403 is the anglebetween the reference line and the interocular line projected on the HUDplane. Also, as part of step 405, a location 459 of the viewer's eyes isstored, for use in step 455.

For a solid-line HUD marker according to certain embodiments of theinvention, the method can proceed immediately to a step 419, in whichthe HUD marker is displayed with the same (or substantially the same)orientation angle relative to the predetermined HUD reference line ashorizontal orientation angle 403.

For a dashed-line HUD marker according to further embodiments of theinvention, in step 405 interocular distance 108 (FIG. 1) is alsomeasured and stored as an interocular distance value 407.

Techniques for automatically measuring interocular line and interoculardistance are well-known, such as via a digital camera with head posedetection software that can identify the positions of the viewer's eyes.From the digital camera image thus obtained, both viewer's interoculardistance 108 (FIG. 1) and orientation of the interocular line 203 can bemeasured automatically. According to an embodiment of the invention, adigital camera is positioned to image the viewer's face from thedirection in which the viewer would look to see the HUD. Then, shouldthe viewer's head be turned, the applicable foreshortening of theviewer's interocular distance will be imaged, so that no correction toaccount for the foreshortening is needed.

For a dashed-line HUD marker according to other embodiments of theinvention, the method proceeds to a step 409 in which a parallaxdisparity distance 155 (FIG. 1) is calculated as previously described,based on triangular measurements such as those illustrated in FIG. 1.Then the measured value of parallax disparity distance 155 is stored asa parallax disparity distance value 411. This step requires the range tothe target, or assumes this range is large enough to be treatedoptically as infinity.

In a step 413 dash-to-dash distance 351 (FIG. 3C) is calculated and thevalue is stored in a dash parameter storage 417. In a step 415, theremaining dash parameters, namely dash segment length 353 and dash gaplength 355 (FIG. 3C) are set and similarly stored in dash parameterstorage 417.

For a dashed line marker, step 419 completes the process by displayingthe marker on the HUD, using the dash parameters stored in dashparameter storage 417 as well as horizontal orientation angle 403.

A further embodiment of the present invention provides a computerproduct for performing the foregoing method, or variants thereof.

A computer product according to this embodiment includes a set ofexecutable commands for performing the method on a computer, wherein theexecutable commands are contained within a tangible computer-readablenon-transient data storage medium including, but not limited to:computer media such as magnetic media and optical media; computermemory; semiconductor memory storage; flash memory storage; data storagedevices and hardware components; and the tangible non-transient storagedevices of a remote computer or communications network; such that whenthe executable commands of the computer product are executed, thecomputer product causes the computer to perform the method.

In this embodiment, a “computer” is any data processing apparatus forexecuting a set of executable commands to perform a method of thepresent invention, including, but not limited to: a processor; embeddedcontroller; or any other device capable of executing the commands.

Apparatus

FIG. 5 illustrates a conceptual block diagram of apparatus 501 forcontrolling a head up display system 503 to display a head up displaymarker according to embodiments of the invention.

A head up display (HUD) system 503 displays information and markers fora viewer viewing a far-focus visual target (not illustrated). A unit 505identifies and locates the apparent visual position and extent of thevisual target relative to the HUD seen by the viewer.

A digital camera 509 faces the viewer and captures an image of theviewer's face, which is recognized as a human face by a face detectionunit 511. An eye location unit 513 locates the positions of the viewer'seyes, and outputs the positions to horizontal orientation unit 515,which determines a horizontal orientation angle for a one-dimensionalmarker. In a related embodiment of the invention, horizontal orientationunit 515 measures the angle of the viewer's interocular line relative toa reference line in the HUD via digital camera 509. Horizontalorientation unit 515 outputs the horizontal orientation angle to aone-dimensional marker display generator 507.

One-dimensional marker display generator 507 then inputs commands to HUDprojection system 503 to display a one-dimensional marker at one or morepositions within the HUD according to the location and extent of thevisual target as determined by visual target identifier 505, wherein theorientation angle of the one-dimensional marker relative to thereference line in the HUD is substantially equal to the horizontalorientation angle, which in the related embodiment of the invention, aspreviously detailed, is the orientation of the viewer's interocular linerelative to the reference line.

In an additional embodiment of the invention, an interocular distancemeasuring unit 517 obtains eye position data from eye location unit 513and sends the viewer's interocular distance to a parallax disparitycalculator 519. The calculated parallax disparity is then sent fromparallax disparity calculator 519 to a dash parameter calculator 521,which computes dash parameters, as previously described, including thedash-to-dash distance. When dash parameter calculator 521 sends the dashparameters to marker display generator 507, marker display generatorsends commands to HUD projection system 503 to display a dashedone-dimensional marker at one or more positions within the HUD accordingto the location and extent of the visual target as determined by visualtarget identifier 505.

What is claimed is:
 1. A method for displaying, to a viewer, a marker ona head up display (HUD), the method comprising: determining a horizontalorientation angle relative to a predetermined reference line of the HUD;and displaying a one-dimensional marker having a marker orientationangle measured relative to the predetermined reference line of the HUD,wherein the marker orientation angle is substantially equal to thehorizontal orientation angle.
 2. The method of claim 1, wherein thedetermining the horizontal orientation angle is performed by measuring aviewer interocular line relative to a predetermined reference line ofthe HUD.
 3. The method of claim 1, wherein the one-dimensional marker isa solid line.
 4. The method of claim 1, wherein the one-dimensionalmarker is a dashed line.
 5. The method of claim 4, wherein the viewer isviewing a visual target having a parallax disparity with respect to thedashed line, and further comprising: calculating a parallax disparitydistance of the parallax disparity; and setting a marker dash-to-dashdistance of the dashed line to a unit fraction of the parallax disparitydistance.
 6. Apparatus for controlling a head up display system, fordisplaying, to a viewer, a marker on a head up display (HUD), theapparatus comprising: a visual target identifier, for identifying aposition of a visual target which the viewer is viewing through the HUD;and a marker display generator for controlling the HUD system to displaya one-dimensional marker according to the position of the visual target,wherein an orientation angle of the one-dimensional marker relative to areference line of the HUD is substantially equal to an orientation of aninterocular line of the viewer's eyes relative to the reference line. 7.The apparatus of claim 6, further comprising: a digital camera, forimaging the viewer's face; and an eye location unit, for locatingrespective positions of the viewer's eyes and determining an interocularline orientation relative to a reference line of the HUD wherein thehorizontal orientation angle unit determines the horizontal orientationangle according to the interocular line.
 8. The apparatus of claim 7,wherein the one-dimensional marker is a solid line.
 9. The apparatus ofclaim 7, wherein the one-dimensional marker is a dashed line.
 10. Theapparatus of claim 7, further comprising: an interocular distance unit,for measuring an interocular distance of the viewer; a parallaxdisparity calculator for calculating the a parallax disparity distance;and a marker dash parameter calculator for calculating a dash-to-dashdistance of the dashed line according to the parallax disparitydistance.
 11. A computer product for a computer controlling a head updisplay system, for displaying, to a viewer, a marker on a head updisplay (HUD), the computer product comprising a set of executablecommands contained within a tangible computer-readable non-transientdata storage medium, such that when the executable commands of thecomputer product are executed, the computer product causes the computerand the head up display system to: determine a horizontal orientationangle relative to a predetermined reference line of the HUD; and displaya one-dimensional marker having a marker orientation angle measuredrelative to the predetermined reference line of the HUD, wherein themarker orientation angle is substantially equal to the horizontalorientation angle.
 12. The computer product of claim 11, furthercomprising additional executable commands, which, when executed, causethe computer and the head up display system to measure an orientation ofa viewer interocular line relative to a predetermined reference line ofthe HUD and to determine the horizontal orientation angle according tothe viewer interocular line.
 13. The computer product of claim 11,wherein the one-dimensional marker is a solid line.
 14. The computerproduct of claim 11, wherein the one-dimensional marker is a dashedline.
 15. The computer product of claim 14, wherein the viewer isviewing a visual target having a parallax disparity with respect to thedashed line, and further comprising additional executable commands,which, when executed, cause the computer and the head up display systemto calculate a parallax disparity distance of the parallax disparity;and set a marker dash-to-dash distance of the dashed line to a unitfraction of the parallax disparity distance.